Papers with cross-modal reasoning
Filter-And-Refine: A MLLM Based Cascade System for Industrial-Scale Video Content Moderation (2025.acl-industry)
Copied to clipboard
Zixuan Wang, Jinghao Shi, Hanzhong Liang, Xiang Shen, Vera Wen, Zhiqian Chen, Yifan Wu, Zhixin Zhang, Hongyu Xiong
| Challenge: | Effective content moderation is essential for video platforms to safeguard user experience and uphold community standards. |
| Approach: | They propose a method to transform a generative MLLM into a multimodal classifier using minimal discriminative training data. |
| Outcome: | The proposed method improves F1 score by 66.50% over traditional classifiers while requiring only 2% of the fine-tuning data. |
ExCL: Extractive Clip Localization Using Natural Language Descriptions (N19-1)
Copied to clipboard
| Challenge: | Prior approaches to retrieving clips within videos based on a given query are inefficient and text-clip similarity driven ranking-based approaches are far more complicated. |
| Approach: | They propose an extractive approach that extracts the start and end frames by leveraging cross-modal interactions between the text and video to generate a joint representation. |
| Outcome: | The proposed approach significantly outperforms state-of-the-art on two datasets and has comparable performance on a third. |
“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns? (2026.acl-long)
Copied to clipboard
Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li, Shouling Ji
| Challenge: | Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. |
| Approach: | They propose a multimodal pun generation pipeline and a model to evaluate their understanding of puns. |
| Outcome: | The proposed benchmark improves the understanding of multimodal puns by 16.5% in the F1 test. |
Unveiling Fake News with Adversarial Arguments Generated by Multimodal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for detecting fake news rely on neural networks to learn latent feature representations with limited real-world understanding. |
| Approach: | They propose a method that leverages Multimodal Large Language Models for fake news detection that introduces adversarial reasoning through debates from opposing perspectives. |
| Outcome: | The proposed method significantly outperforms state-of-the-art methods on four fake news detection datasets. |
Unifying Text, Tables, and Images for Multimodal Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal question answering rely on single-modal or bi-modal models, which limit their ability to integrate information across all modalities. |
| Approach: | They propose a framework that unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. |
| Outcome: | The proposed framework unifies three input modalities into a text-to-text format using position-enhanced table linearization and diversified image captioning techniques. |
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Multimodal Large Language Models (MLLMs) are predominantly trained on consistent visual-textual inputs, leaving open the question of whether they can handle semantic mismatches in layout-rich content. |
| Approach: | They propose to use multimodal inconsistency reasoning to assess MLLMs' ability to reason about semantic mismatches in webpages, presentation slides, and posters. |
| Outcome: | The proposed model outperforms open-source models in detecting inconsistencies in webpages, presentation slides, and posters while remaining vulnerable to inconsistent errors. |
WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts (2025.findings-acl)
Copied to clipboard
Negar Foroutan, Angelika Romanou, Matin Ansaripour, Julian Martin Eisenschlos, Karl Aberer, Rémi Lebret
| Challenge: | Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). |
| Approach: | They propose a benchmark for evaluating cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages . they evaluate 12 vision-language models that achieve 70% accuracy when provided with direct context . |
| Outcome: | The proposed benchmark evaluates models with high accuracy over tables and charts extracted from 4,000 Wikipedia pages . proprietary models achieve 70% accuracy when provided with direct context, but open-source models perform worse when retrieval from long documents is required. |
Fast Retrieval and Slow Reasoning for Explainable Multimodal Sentiment Analysis (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal sentiment analysis methods rely on holistic fusion . such strategies introduce redundant information and obscure the decision process . |
| Approach: | They propose an interpretable framework that decomposes multimodal sentiment modeling into two cooperative pathways. |
| Outcome: | The proposed framework achieves competitive performance, higher efficiency, stronger robustness to noise, and clearer decision transparency than existing holistic fusion methods. |
UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets (2025.emnlp-main)
Copied to clipboard
Pengyu Wang, Shaojun Zhou, Chenkun Tan, Xinghao Wang, Wei Huang, Zhen Ye, Zhaowei Li, Botian Jiang, Dong Zhang, Xipeng Qiu
| Challenge: | Existing datasets address understanding and generation in isolation, limiting the performance of unified vision large language models. |
| Approach: | They propose a dataset that facilitates mutual enhancement between multimodal understanding and generation. |
| Outcome: | The proposed framework integrates diverse visual and textual inputs and outputs, enabling comprehensive cross-modal reasoning and precise text-to-image alignment. |
ConSensus: Multi-Agent Collaboration for Multimodal Sensing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. |
| Approach: | They propose a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. |
| Outcome: | The proposed framework matches or exceeds debate methods on multimodal sensing benchmarks while achieving 12.7 times reduction in token cost. |
Probing Audio-Visual Reasoning in Multimodal Language Models through the Lens of Audio (2026.acl-long)
Copied to clipboard
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue
| Challenge: | Recent multimodal large language models lack robust audio-visual integration ability and performance on DeafTest is highly correlated with AV-Odyssey accuracy. |
| Approach: | They propose a benchmarking tool that integrates audio-visual reasoning with audio-video cues to infer solutions. |
| Outcome: | The proposed model performs well on DeafTest, but lacks audio perception in simple audio tasks. |
MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing RAG solutions for large language models are limited by context windows limiting their ability to process long-form, domain-specific content. |
| Approach: | They propose a multimodal knowledge graph-based RAG that enables cross-modal reasoning . their method incorporates visual cues into the construction of knowledge graphs, retrieval phase, and answer generation process . |
| Outcome: | Experimental results show that the proposed approach outperforms existing approaches on textual and multimodal benchmarks. |